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Abstract 

The IMS Learning Design specification provides a potential means for capturing 
units of instruction in a machine-readable, consistent way. However, in order for 
the IMS Learning Design specification to be used widely by educators and 
instructional designers for whom it is intended, we will need effective ways for 
users to contribute to, access and adapt the repositories where reusable learning 
designs are collected and stored. This paper describes a project conducted to 
develop and test a prototype search model for learning design repositories. We 
argue for development of a controlled vocabulary to describe and label learning 
designs. In this way, designs can be accessed according to a variety of 
pedagogical, as well as topical criteria specific to the instructional purposes and 
perspectives of the user. 

Resume: La caracteristique de conception de scenarios d'apprentissage de la 
norme IMS permet de saisir des unites d'enseignement uniformes et lisibles par la 
machine. Toutefois, pour que cette caracteristique soit utilisee couramment par les 
educateurs et les concepteurs de materiel d'enseignement auxquels elle s'adresse, 







nous devrons mettre en place des methodes efficaces permettant aux utilisateurs 
de contribuer aux logitheques, d'y acceder et de les adapter lorsque les scenarios 
d'apprentissage reutilisables sont recueillis puis emmagasines. L'article decrit un 
projet portant sur la conception et la mise a I'essai d'un modele de recherche 
pilote pour les logitheques de scenarios d'apprentissage. Nous penchons vers 
I'elaboration d'un vocabulaire selectionne « axe sur I'utilisation » plutot qu'un 
vocabulaire selectionne « axe sur la theorie » afin de decrire et d'identifier les 
scenarios d'apprentissage. Ainsi, on pourra acceder aux scenarios au moyen de 
divers criteres lies a la pedagogie et aux sujets propres aux objectifs 
d'enseignements et aux perspectives de I'utilisateur. 

Introduction 

The IMS Learning Design [LD] specification (IMS Global Learning Consortium, 2003) offers 
tremendous potential for providing sharable, interoperable resources for instructors and 
instructional designers, in the form of reusable designs for units of learning that meet a wide 
range of instructional goals. For this power to be realized, however, more effort needs to be 
focused on developing conceptual documentation and tools for instructors to use in accessing, 
adapting, and in further developing such reusable descriptions and designs. 

This paper describes our part of a collaborative project, involving several Canadian university 
partners, investigating the potential of IMS LD in a post-secondary environment. At the 
University of Waterloo, we focused on issues concerning the development of pedagogical 
language, or vocabulary, to describe units of instruction in the context of the IMS LD 
specification. We created a prototype search interface and tested various search criteria that we 
hypothesized would be meaningful to users from a wide range of disciplinary and pedagogical 
perspectives. The results of our user test confirmed wide variability in opinions and perspectives 
among faculty in terms of how learning design repositories should be organized and subsequently 
searched. Ultimately, one goal of projects attempting to facilitate the implementation of IMS LD 
specification should be to develop a controlled vocabulary that will allow users (both faculty and 
instructional designers) to use repositories to inform themselves about generic learning designs 
and to access specific instantiations of them within a variety of content domains. 

Task Definition and Methodology 

The IMS LD specification allows for capturing learning designs in formal XML machine- readable 
terms. There are specified fields used to express descriptions of activities and their sequences, 
the roles played by students, instructors and other staff, and the resources, services and 
facilities required for completing each activity or set of activities. From a user's point of view, 
though, it is important to distinguish between this use of the term "Learning Design" (i.e., as a 
formal XML expression) and "learning designs," more generally. 

Our investigation focused on the use of the IMS Learning Design specification in the development 
and use of repositories of learning designs, as an extension of the developments related to 
learning object repositories. Repositories of learning objects have been attracting growing 
interest as a means for sharing instructional resources (Campbell, 2003; Koper, Pannekeet, 
Hendriks, & Hummel, 2004; see also MERLOT, 2004, and CLOE, 2004). Learning object repository 
projects such as eduSOURCE (2004), Education Network Australia (EdNA, 2004), and Learning 
Online Network with CAPA (LON-CAPA, 2004) have made significant progress in developing 



models for search and retrieval of learning objects, both within and across repositories, and in 
developing and testing metadata standards. 

Ultimately, our goal is to create repositories that contain both objects and explicit 
representations of the learning designs, which provide their educational rationale to enable 
sharing of instructional knowledge. A learning design, by our definition, is much like a lesson plan 
that makes use of learning objects as resources for completing activities. A learning design 
represents a unit of study, which necessarily contains the associated learning objective(s), the 
sequence or sequences of activities to be engaged in, a description of the learning environment, 
who plays what roles in the learning activities, and what resources and services are used to 
complete the learning activities. As noted by Koper (2001), "In practice you see units of study in 
all types, sorts, and sizes: a course, a study program, a workshop, a practice, a lesson could all 
be considered to be a unit of study" (p. 3). While in some instances learning objects may match 
our definition of a learning design, most do not. They are designed to be used within many 
different kinds of lessons, in many instructional contexts, and for many different instructional 
purposes. 

The IMS LD specification offers a consistent, machine-readable way to describe learning designs. 
Adaptations to existing repositories will be required in order to store and search for Learning 
Design information. The generation and implementation of a "controlled vocabulary" for 
identifying and tagging those designs within repositories will further facilitate their sharing and 
re-use. A controlled vocabulary is critical to the retrieval of web-based resources, as noted by 
Fast, Leise, and Steckel (2001): 

The most effective communication occurs when all parties involved agree on the meaning of the terms being used....When we 
converse, we speak in “natural language.’’ This is language in all its raw, rich, gooey glory. When we organize our information 
and label it however, there is so much richness, variance, and confusion in terminology that we often need to impose some 
order to facilitate agreement between the concepts within the site and the vocabulary of the person using it. 

This order can come through a controlled vocabulary. Warner (2002) defines a controlled vocabulary (C V) as “organized lists 
of words and phrases, or notation systems, that are used to initially tag content, and then to find it through navigation or 
search.” (1-2) 

Currently, there is an extremely large and yet unknown number of potentially useful learning 
designs and virtually a limitless number of ways in which they can be labelled and classified. The 
identification of a comprehensive set of generic, sharable learning designs that lend themselves 
to application across instructional contexts and disciplines is, in itself, a daunting task. We 
believe that a critical mass must be identified before a controlled vocabulary can begin to be 
defined. 

Because the potential list of learning designs or generic, sharable lesson plans is virtually 
limitless, it is necessary to explore with potential users their conceptualizations of what kinds of 
designs they envision sharing, and how these might be categorized in a collection. The 
categorization they envision will depend on the purposes users have for the designs within their 
own teaching. What are the instructional challenges they face, and how do they conceive of 
lesson plans that would meet those needs? They may think of finding lessons that address a 
specific topic or concept, or that offer an opportunity for learners to consider the ideas of others, 
or to confront common misconceptions, or to learn to solve authentic problems in their subject 
area while working in a team context. They may see themselves using lessons for brief practice 
activities or for extended projects over weeks of class time. In a usage-centred approach, these 



and other purposes will define the ways in which learning designs are described and classified 
within a design repository. The vocabulary used to refer to the designs needs to be controlled, 
but also cross-referenced so that users can access them with a variety of pedagogical purposes. 

We found the work of the Learning Designs Project (2003), commissioned by the Australian 
Universities Teaching Committee to be an excellent start for this task, in part because the 
designs they included were "proven" through evaluation and in part because they are described 
in practical, purposeful pedagogical terms. Although their project and its outcomes did not 
address the IMS LD specification, their aims in terms of identifying sharable, generic designs for 
units of learning were very similar to ours. Their definition of what a learning design encompasses 
also agreed with ours and with that used by the IMS group. The following excerpt from their Final 
Report illustrates how readily their conceptualizations can be transferred and tested in another 
context. 

This Australian University Teaching Committee (AUTC) project aimed to produce generic/reusable learning design 
resources to assist academics to create high quality, flexible learning experiences for students. This was achieved by: 

1. Identifying high quality learning designs used in higher education; 

2. Selecting those suitable for redevelopment in the form of reusable software, templates and/or generic 
guidelines; and 

3. Developing these reusable resources and making them accessible from a central web site. 

The term "learning design" refers to a variety of ways of designing student learning experiences, 
that is, the sequence of activities and interactions. The scope of a learning design may be at the 
level of a subject/unit or subject/unit components. This project focuses on learning designs 
implemented with the use of ICT and how flexible learning opportunities for students can be 
afforded through the use of such technologies. The composition of a learning design, particularly 
when ICT mediated, has been informed by the work of Oliver (1999) and Oliver and Herrington 
(2001). Thus, for the scope of this project, a learning design comprises three key elements: the 
content or resources learners interact with, the tasks or activities learners are required to 
perform, and the support mechanisms provided to assist learners to engage with the tasks and 
resources ( 4). 

The web site created through the AUTC Learning Designs project contains descriptions of 32 
learning designs, each of which is described in generic [content-independent] terms and then 
instantiated with specific subject matter and instructional contexts. They have grouped their 
designs and exemplars into five categories, each loosely described as a "learning focus." 
However, these groupings are somewhat arbitrary, are not mutually exclusive, and would need to 
be cross-referenced with other search vocabulary if they were to be used effectively for 
searching within a learning design repository. An advantageous feature of the AUTC 
categorizations of learning designs, from our perspective, is that they are usage-centred. 
Specifically, the five "learning focus" categories: Collaborative, Concept/Procedure Development, 
Problem-Based Learning, Project/Case Study, and Role Play, are reflective of well-known learning 
activities or approaches that instructors might consider when designing learning activities, 
lessons, or assignments. 

We tested several of the learning designs from the Learning Designs Project web site, as well as 
some designs that have been identified and used effectively at the University of Waterloo, in the 
context of identifying potential search models and terms. The objectives we identified for this 
project were to obtain input from faculty members and instructional designers on: 



• The potential usefulness of repositories containing learning designs, 

• Their conceptual models for cataloguing and searching for learning designs, and 

• Vocabulary to be used in searching for and accessing learning designs. 

An assumption underlying this investigation was the importance of being able to link the learning 
design representations to IMS LD scripts to facilitate interoperability for learning design 
implementations. With this assumption in mind, it was essential for us to complete the following 
tasks as a first step in reaching our objectives: 

• Identify and describe exemplars of learning designs that can be captured in terms of valid IMS LD 
scripts 

• Document the pedagogical principles and supporting evidence for these learning designs 

• Define teacher-friendly pedagogical search terms for the LD exemplars 

Criteria for Selecting Sample Learning Designs for Inquiry 

A number of parameters guided our selection of learning designs for illustration and obtaining 
user input. As a first criterion, learning designs were selected that generally fit the definitions in 
the IMS LD specification in terms of their scope, level of granularity, purposes and components. 
By this criterion, we selected full-featured learning activities that included learning objectives, 
instructions and identification of instructor and student roles. We also included designs that 
represented a range of instructional scenarios such as duration of engaged learning time and 
learner interactions, as well as varied subject matter and learner groupings. 

A second criterion was that the learning designs involved multi-person scenarios such as small 
group activities rather than being limited to individualized or whole class learning activities. This 
feature is a major differentiation of the IMS LD specification from the IMS Simple Sequencing 
specification. 

A third criterion used to identify learning designs for this user test was based on the assumption 
that instructors or instructional designers would benefit from designs illustrated by content 
across a range of subject areas. This is a key advantage for a repository containing both learning 
designs and objects; even if there is not a specific learning object that addresses the 
instructional challenge encountered by an instructor, there may be a relevant design addressing 
a similar challenge but with other subject matter. We therefore intended that our samples would 
represent a wide range of disciplines, instructional situations and learning objectives. It is 
expected that a learning design repository would include such a range, and that users from all 
disciplinary backgrounds would be able to access ideas and examples across these subject 
matter and pedagogical boundaries. 

We anticipated that faculty would access a learning design repository when they are faced with 
an instructional challenge or bottleneck, for which they had no immediate solution. In order for 
the repository to be usage-centred for faculty in this sense, it would be important that the 
learning designs address a specific instructional challenge or objective. Therefore, this was our 
fourth criterion for inclusion. 

The learning designs chosen also needed to represent a wide range of exemplars of the concept 
of a learning design, in terms of teaching and learning applications. As a result, it was important 
that they vary considerably in scope, size and granularity. To meet this criterion we included 
designs that ranged from brief activities within a "class" session to extended sets of activities 



spanning a semester-long course. 


Finally, it was critical for us that the designs chosen for this inquiry should be grounded in 
educational/pedagogical literature and/or evaluation reports. References to evidence that 
supports the validity of the learning designs is considered an essential aspect of a future 
repository, to allow users to assess their merits. We believe that repositories of learning objects 
and learning designs can thus serve as exemplars of scholarship applied to teaching and learning. 

Conceptual Model for Learning Design Repositories 

We found the kinds and structure of learning design information in the Learning Designs Project 
group site potentially useful because they included generic, content-independent designs and 
domain-specific instantiations of these generic designs. In this way, the designs could be adopted 
for use in a wide variety of instructional contexts, much as a template could, but the user could 
also see illustrations of how other instructors have used the designs in specific situations. In the 
context of a learning design repository, we envision that each record would contain one generic 
design or template and several domain-specific examples for each learning design. To be most 
broadly useful, the specific examples should reflect as many different disciplines or subjects as 
possible. 

Figure 1 illustrates our initial concept of a learning design repository access model, including the 
information that should be provided for each learning design record. As shown in Figure 1, the 
instructor would approach the repository with an instructional scenario in mind. The search terms 
used would reflect aspects of that instructional scenario. Included in each record within the 
repository would be title, descriptions and explanations of learning designs, links to instantiations 
of examples ( exemplars ) and/or to templates for generic designs, and references to evaluations 
or research literature supporting the design. Th eexplanation portion of each record would include 
descriptions of appropriate support mechanisms to assist learners in engaging with the tasks and 
resources. Also included in each record there would necessarily be a link to the Learning Design 
XML. 


Information Related to Repository Records for Learning Designs 

Explanations. An explanation includes a definition and description of the key features of a 
learning design that addresses specific teaching/learning challenges, in terms of its pedagogical 
elements. (Pedagogical elements of a learning design include roles, activities, resources, 
environment and workflow.) The explanation outlines, in the most generic form possible, the 
essential structure and steps involved in the learning design. The explanation may include 
scenarios of use. 

Exemplars. Exemplars are illustrative examples of learning designs as they are applied in the 
context of specific teaching/learning situations within various content domains. They may, in 
some cases, be existing learning objects found within repositories, or they may be illustrative 
descriptions of teaching/learning situations. As noted above, there should be multiple examples 
for each type of learning design, as applied across varied subject area domains. This allows 
faculty and designers to envision how their own instructional challenges might be addressed, 
after seeing how similar challenges have been addressed by others in their own and other 
disciplines. 



INSTRUCTIONAL SCENARIO 



Generic Learning Design 


Exemplar / Instance 


- Title 

- Explanations 

- Evaluation 

- Design Resources 

- Links to Exemplars / 
Instances 

- Link to LD XML 


- Title 

- Explanation 

- Evidence 

- Evaluation 

- Design Resources 

- Link to generic design 

- Link to Run time 

- Link to LD XML 


Figure 1. Potential search model for learning design repository records, identifying information to 
be included with each record. 


Evidence from the Literature. The evidence component of the LD Vocabulary refers to the 
rationale for how a learning design will work, including references to the research literature in 
education. It applies necessarily to the design and, in some cases, to an instance of the design. 
It is important to include research evidence that supports the pedagogical validity of the designs, 
to facilitate user assessment of designs they are considering, and to help users from various 
disciplines to become and remain current in the relevant educational research literature. 







Evaluation. Evaluation is related to specific instantiations of the design within a given 
instructional context. This may take the form of peer reviews, learning outcome studies, student 
evaluations or other forms of evaluation that support the use of the learning design in a particular 
case. Evaluations from other instances in which a design has been applied can contribute to the 
supporting evidence for a learning design that is newly added to a repository. 

Technical Resources. The technical resources section of each record contains a link to the 
actual Learning Design XML record, and any associated top-level meta-data, which also appears 
in XML. Both XML files can be downloaded independently, or as part of an IMS Content Package. 


Toward a Controlled Vocabulary of Pedagogical Search Terms 

We hypothesized that instructors and designers will be guided in their search for learning designs 
by the following types of information (Carey, Swallow, & Oldfield, 2000): 

• Content [subject area, topic, learning objectives]; 

• Audience [age range, grade level, prerequisite knowledge or mastery level, perhaps motivation or goals 
for learning]; and 

• Process [learner tasks, cognitive activities required, instructional approach]. 

In approaching a search from the perspective of pedagogical needs, we suggest one potential 
type of search criterion, referred to as Educational Rationales [ER] for learning designs (see 
Figure 2). Educational Rationales identify and describe the cognitive activities learners are 
required to engage in to complete a given unit of instruction. We believe that these descriptors 
will allow informed selection and reuse of learning designs and also will be general enough to 
reveal similarities between designs. For example, users should be able to search for a variety of 
designs that require students to apply theory in practice or to collaborate to accomplish part of 
the learning task. 

The ER search option represents a primary example of what a controlled vocabulary might 
consist of. This ER vocabulary refers to a set of 21 learning processes that capture specific 
instructional approaches. Each of the learning processes identified in this list of options has been 
shown to be valuable in certain learning contexts (Carey et al., 2000). For a learning design 
containing one or more of these ER descriptors, the evidence portion of the repository record can 
cite the relevant research literature. 

Initial User Test of a Hypothetical Search Model 

Our user testing was conducted on the basis of a hypothetical set of steps that faculty or 
designers might take in using learning design repositories. The steps we proposed were: 

1. Identify an instructional situation and challenge, using controlled vocabulary; 

2. Search repositories to find content-independent learning designs and content-specific instantiations of 
those designs; 

3. Preview and assess the merits of results, using references provided [e.g., research literature, peer 
reviews, reports of student evaluations]; 

4. Adopt and adapt design(s) as desired for current instructional needs; 

5. Share adapted or new designs by submitting to repository (Note that review requirements of 
repositories will vary). 


To obtain feedback on this hypothetical process, we conducted a series of interviews. This small 
scale user test involved some demonstration and some description of use case scenarios, 



enabling the participants to imagine how a learning design repository might appear to them, how 
they potentially could search that repository, and how the results might be used in their own 
course design and development. 

For the purposes of the user test we developed a repository containing four learning designs and 
six instantiations of designs within specific content domains. We then created a prototype user 
interface, which offered access to the repository, using a simulated search engine and specified 
search criteria. 

The hypothetical Learning Design Repository search interface is shown in Figure 2. Participants 
were able to search our "repository" for any of the ten records, and examine the results that 
were obtained. 

Participants. Presentation Model, and Procedures 

We interviewed four faculty members and two instructional design experts at the University of 
Waterloo. The faculty members included two from Engineering, one from Peace and Conflict 
Studies, and one from Modern Languages. 

Each participant spent one hour in a one-on-one interview session. They were introduced to the 
concepts of learning design re-use and repositories through a presentation and demonstration. To 
demonstrate a learning design labelled Predict, Observe, Explain, as it can be instantiated in an 
online environment, we included a run-time template and domain-specific (physics) exemplar, 
available from the Learning Designs Project (2003) web site. 

Each participant was then asked to imagine that they were teaching a course in one of the 
subject areas represented in our repository, and in which they encountered an instructional 
challenge that we specified. This hypothetical scenario was presented to them in order to 
demonstrate how the search model could work, and to observe their reactions to the search 
criteria and vocabulary we suggested. Participants were able to search the repository using 
cross-referenced search terms, such as discipline, delivery method and ER descriptor. The 
Results page that would be shown for a search using none of the search criteria options displays 
a list of all the titles of learning designs in the repository (see Figure 3). An example of the 
Results page for the learning design titled Predict, Observe, Explain is presented in Figures 4a 
and 4b. 



Search for l earning designs using any or all of (he following criteria: 
Keywords: | [Optional] 


Discipline: | Any 

Delivery Method: 

<* Any 

C Face to Face 
C Online 
r Blended 

Educational Rationale: 

What do you want your learners to do in this Unit of Learning? 

[Check all that apply] 

r Anchor new knowledge in authentic contexts 
r Set a goal to solve a non-trivial case or problem 
r Develop motivation to perform tasks and understand knowledge 
r Apply theory in practice 
r Employ multiple styles of learning 
r Customize the learning agenda 
r Monitor comprehension and adjust learning strategies 
r Adapt task difficulty to match needs and capabilities 
r Engage in expository or teaching activities 
r Use trial and error to discover something new 
r Collaborate to accomplish part of the learning task 
r Engage in self-evaluation 
F Reflect on the learning process 
r Confront and resolve misconceptions 
r Extrapolate beyond the information provided 
r Relate new knowledge to prior knowledge 
r Examine new knowledge from different perspectives 
r Differentiate knowledge types e.g., heuristics, context-dependent 
r Integrate new knowledge 
r Elaborate new knowledge 
r Think critically about new knowledge 

Search Now | 

Top 


Figure 2. Hypothetical learning design repository search interface. 

We asked participants to think aloud and ask questions as they explored the user interface and 
considered the usefulness and potential applications of the search criteria and vocabulary, as well 
as the results that were shown. The following general questions guided the interviews: 

1. If a learning design repository such as this one existed, is it something that you would use? Do you 
think re-usable learning designs make sense in your discipline? 

2. Does this search interface make sense to you? In what other ways would you want to be able to 
search for learning designs? 

3. What comments and suggestions do you have about the search vocabulary included here? Are these 
useful terms for you? If so, are there other terms you would add to the list? If not, what other kinds of 




terms would make more sense? 


Return to Search 


1. Task Discussion - Stereoty pin g 

Stereotyping Discussion Task 

Online - Philosophy 

2. Observe, Represent Collaborate, Compare, and Refine - VisChem 

VisChem: Visualizing Chemical Structures and reactions at the Molecular Level to Develop a Deep Understanding of 
Chemistry Concepts 

Blended - Chemistry 

3. Observe, Represent, Collaborate, Compare, and Refine - VisChem 

VisChem: Visualizing Chemical Structures and reactions at the Molecular Level to Develop a Deep Understanding of 
Chemistry Concepts 

Online • Chemistry 

4. Thesis Proposal - Biology 

Thesis Proposal Assignment in Molecular Genetics of Plant Development 

Blended - Biology 

5. Predict Observe, Explain - Hammer and Feather 

Predict, Observe, Explain: Hammer and Feather Problem 

Blended - Physics 

6. Critical Analysis - Environmental Studies 

Critical Analysis Program 

Online - Environmental Studies 

7. Predict Observe, Explain - Template 

Predict. Observe, Explain Template 

N/A-N/A 

8. Thesis Proposal - Template 

Thesis Proposal Assignment 

N/A-NIA 

9. Observe, Represent Collaborate, Compare, and Refine - Template 

Visualizing scientific phenomenon at the contrete, observable level, the non-visible level, and the symbolic or quantitative 
level 

N/A-Hi’A 

10. Task Discussion - Template 

Discussion Task Template 

N/A-WA 


las 


Figure 3. Sample search results page. 




Return to Search 

Explanation Evidence Evaluation Exemplars Technical Resources 

Name: 

Predict, Observe, Explain 

Title: 

Hammer and Feather 

Discipline: 

Physics 

Delivery: 

Blended 

Source: 

AUTC 

Type: 

specific 

Prerequisites: 

1. None. 


1. To articulate, justify and critically reflect on self and partner's 

Learning 

physics preconceptions and become aware of alternative 

Objectives: 

conceptions. 

View It 


Explanation 


l. Students arc introduced to predict, observe, explain strategy and then form pairs. 

2. Pairs discuss and predict outcome for Hammer and Feather scenario. Prediction, its rationale and 

group member’s level of commitment to prediction arc recorded. 

3. Paris observe scenario outcome shown on video within CD-ROM. 

4. Groups of pairs explain any differences between their predictions and observations. Group 

discussions are 

faciliatcd by the instructor. Explanation is recorded. 

5. Class debrief: Facilitated whole-class discussion of student predictions and explanations about 

what happens when the astronaut drops the hammer and feather. 


Evidence 


1. Motivate - Develop motivation to perform tasks and understand knowledge 

Student motivation can be achieved by attention-getting, relevance-producing, confidence- 
building, and satisfaction-generating strategics (Keller, 1987). An adult's motivation to learn 
depends largely upon how relevant the material is to their job or personal life (Knowles, 1984). 
Students must be willing and able to leant as a result of the experiences and contexts provided by 
instruction (Bruner, 1966). 

2. Relate - Relate new knowledge to prior knowledge 

Learning activities should enable learners to make use of their prior experiences (Cross, 1981). 
Related prior knowledge is the most important factor in mediating the acquisition of knowledge 
(Park & Hannafin, 1993). 


Figure 4a. Hypothetical record for the Predict, Observe, Explain learning design. 




3. Collaborate - Collaborate to accomplish part of the learning task 

Learners benefit from social interaction and collaboration (Lave & Wenger, 1990; Vygotsky 1978). 

4. Monitor - Monitor comprehension and adjust learning strategies 

Learners need to be able to monitor their comprehension, and choose appropriate learning 
strategies, in order to use exploratory media effectively. Prompts and self-checks can help 
learners monitor learning and choose strategies (Park & Hannafm, 1993). 

5. Perspectives - Examine new knowledge from different perspectives 

Content should be represented in multiple ways, with many, diverse examples. (Spiro, Feltovich, 
Jacobson, & Coulson, 1992). 

6. Misconceptions - Confront and resolve misconceptions 

Learners can compartmentalize experiences, allowing naive and ritual patterns of 
misunderstanding to persist after instruction (Perkins & Simmons, 1988). Sources of 
misconception include applying everyday meanings for technical terms and simplifying the 
internal structure of a concept (Laurillard, 1993). 

7. Integrate - Integrate new knowledge 

The nature of ill structured domains requires that learners interconnect knowledge. (Spiro, 
Feltovich, Jacobson, & Coulson, 1992). Learners need to acquire integrate declarative and 
procedural knowledge, heuristics, and control process, in order to become successful problem 
solvers (Schoenfeld, 1985). The more information is processed during learning, the more it will 
be retained and remembered (Craik & Lockhart, 1972). 


Evaluation 

i. None. 

Exemplars 


This design is an example of: 


Technical Resources 


• Metadata 


• Leamimi Design XML 


Top 


Tod 


Figure 4b. Hypothetical record for the Predict, Observe, Explain learning design. 

Results 

In this section we present an overview of what we learned with respect to the development of a 
Learning Design Controlled Vocabulary and search model. Through the project we also developed 
some recommendations on the details of the current IMS-LD specification. These are presented in 
Appendix B. 

The small sample of users involved in this inquiry proved to be informative and surprisingly 
disparate in their perspectives and suggestions for search mechanisms with which to access 
learning design repositories. They all, however, shared the opinion that re-usable learning 
designs will be an excellent resource for faculty and instructional designers. 


Suggested Vocabulary and Search Model 


A primary aim of this project was to take steps toward the adoption of the IMS Learning Design 
specification through the establishment of a controlled vocabulary for learning designs. Making 
use of feedback from faculty and instructional design specialists, pedagogical literature and web- 
based resources, and our own expertise in educational psychology and online learning 
technologies, we have learned some lessons and also raised some new questions concerning this 
goal. 

In presenting faculty and instructional design specialists with hypothetical scenarios and a search 
interface for accessing sharable learning designs, we found that: 

1. Generally, the notion of being able to access such repositories was positively received; faculty agreed 
that they would find such a resource useful, assuming they could search for learning designs in the 
ways that are meaningful and straightforward for them; 

2. There appeared to be consensus on the perceived need for a glossary of pedagogical search terms, 
with which users could search for learning designs; 

3. The ability to search by ER descriptors using keywords such as Apply, Misconceptions, Reflect, 

Anchor, etc. was seen as a particularly useful mechanism for finding learning designs according to 
pedagogical need; 

4. In addition to a pedagogical vocabulary such as the ER descriptors, the ability to search according to 
and cross-reference among different kinds of criteria (i.e., learning design properties) was seen as 
important; these might consist of subjects and topics, grade level, time required to complete the 
learning tasks involved in a given design, learner groupings (e.g., individual, whole class small 
groups, etc.). 

We came to understand that widespread adoption of the IMS Learning Design specification will 
not be possible until a controlled vocabulary can be agreed upon for use in cataloguing and 
searching for learning designs. Existing literature on controlled vocabularies and ontologies 
supports the notion that searching effectively is not possible unless there is some agreed upon 
terminology within the field to be searched against. The comments from user testers regarding 
the use of a pedagogical vocabulary such as the ER descriptors (Carey et al., 2000) indicated 
that these would potentially be useful kinds of search terms. There was agreement, however, 
that the ER descriptors we presented was limited and incomplete (see Appendix A for specific 
comments). 

Recognizing that users will substitute many different possible pedagogical or cognitive terms and 
phrases for those we listed, we concluded that the establishment of a comprehensive list of 
descriptors and/or ontology of learning designs is an exceedingly labour-intensive and time- 
consuming task. We suggest that existing ontologies or dictionaries of educational terms should 
be linked as a starting place for such a task. For example, the Dictionary of Education: 
Pedagogical Language Usage Server (2004) developed at the University of Michigan provides a 
potential framework for this kind of resource. This web site offers definitions and links to research 
literature for terms found in publications of the American Educational Research Association 
(AERA); it provides multiple terms for similar educational concepts and activities, and allows 
users to access the relevant research literature regardless of the chosen term. Such a resource 
could form the foundation for an ontological engine within the Learning Design repository that 
would be able to handle the mapping of pedagogical search terms. With such a mechanism in 
place, the Learning Designs in the repository would not have to be tagged with all possible search 
terms because the engine would be able to "interpret" what a user might be looking for. 



A complementary approach to what we have described so far would be to develop a user 
interface that operates as a pre-cursor to searching with a controlled vocabulary, and which acts 
like a wizard found in various software applications. Initial steps in this type of approach were 
explored recently at the University of Alberta (see Mattson, 2004). In this scenario, the user 
could provide a free-form response to questions such as, "What type of lesson will this be?" or 
"What do you want to achieve in this activity and how do you think you could get there?" 
Responses to these questions would then be parsed (by the search engine) for key words such 
as the ER descriptor tags we present here, and would provide the user with full definitions and 
descriptions of the pedagogical objectives that the ER tags refer to. This would provide a check 
for the user, to help ensure that the desired kinds of learning designs will be found. The ER 
descriptors could then be linked to relevant exemplary learning designs records. 

Another conclusion reached here was that it would be valuable to have learning objects included 
within repositories of learning designs, or vice-versa. Indeed, one faculty member suggested that 
a useful way to search for learning designs might be according to the learning objects that are 
associated with them. In this way, if a learning object is found to be of particular interest or 
value, users could then search for the learning designs that include that learning object as a 
resource. 

Regardless of whether learning objects and learning designs are housed within the same 
repositories, the links among these different kinds and levels of instructional resources will be an 
important benefit, and also an important design consideration for educators. As noted by Nesbit 
and Winne (2003), access to networked resources by learners working collaboratively on goal 
directed, constructivist learning activities such as problem-based learning, collaborative problem 
solving or project-based learning, changes the cognitive requirements and potential learning 
outcomes in important ways. Quick and easy access to learning resources (e.g., learning objects) 
can either short circuit reasoning processes that are necessitated by less information-rich 
environments or, alternatively, it can offer opportunities for learners to practice other cognitive 
skills, such as evaluation and synthesis of large amounts of information. In the context of their 
learning tasks, ready access to networked resources can also allow the opportunity to address 
problems or projects of larger scope within a given time frame than would be possible if 
resources had to be sought elsewhere. 

Next Steps and Research Directions 
Identification of a Controlled Vocabulary Authority 

The IMS Learning Design specification will not be adopted until there is a reliable and robust way 
to both catalogue and search for Learning Designs. We recommend the identification of an 
authoritative international organization that would be responsible for: 

• the long-term management of the controlled vocabulary 

• making the vocabulary publicly and electronically available 

• maintaining communication with interested contributing parties 

• establishing and maintaining the ontology within the controlled vocabulary 

Establishing a Critical Mass 

Establishing a critical mass of learning designs within a set of repositories is one important step 
in assuring the widespread adoption of IMS Learning Design. In order for searching to be fruitful. 



there needs to be a substantial set of designs to search through. It can be seen that achieving 
this critical mass will allow for refinement of the classification and cataloguing of learning 
designs, which in turn will reinforce the usefulness of the repositories, and further encourage 
widespread adoption. In addition, once a critical mass has been established, we will be able to 
answer such questions such as: 

• Is there some subset of learning designs that are used more often than others? 

• Will analysis of the popularity of certain learning designs reveal trends and patterns in teaching styles? 

• Can we establish a set of pedagogical frameworks, or broad categories of learning designs based on 
such patterns within the repositories? 

Pattern Analysis and Data-Mining 

Given the formal structure and such a potentially large collection of these learning designs, it can 
quite easily be seen how this type of information could yield some very interesting insights into 
pedagogical theory, information science, and cognitive processing. 

There is also a great new potential to identify and label both new and existing pedagogical 
frameworks based on sets of learning designs with similar pedagogical properties. Analyzing the 
type of structured information stored in a learning design repository could also benefit the 
development of a controlled vocabulary by providing a feedback cycle on which terms are 
searched for most often, and which learning design categories are most populated. 

One of the issues we faced throughout this project was the classification of different kinds of 
learning designs. We were faced with making decisions about what level of scope of learning 
design to include in our test repository. Since Learning Designs can be suitable for classes, 
lectures, activities, and anywhere in between, we felt the need to distinguish some sort of level 
to apply to each one we chose. Also at issue is what level of granularity should be applied to 
describe the pedagogical function of a learning design, for example: group-based learning could 
be combined in sequence with self-directed learning and within the group-based learning portion 
there could be a full activity that's best described by a goal-based scenario. There is such a wide 
variety of lesson plans available that picking one as a candidate to be the generic example of an 
overall pattern is extremely daunting. 

One option we can see is the possibility of establishing, identifying, and naming clusters of 
learning design according to their properties and patterns after there is a critical mass of them in 
connected repositories. Given all these possibilities, it can be seen quite easily how this type of 
information would be appealing to information scientists and pedagogical researchers. 
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A ppendix A: Paraphrased Comments from Participants 

1. Comments and Questions Related to Searching 

Usage 

• Trolling for what's new—new idea's new methods 

• Use discipline for browsing for new ideas 

• Use keywords for searching by topic or concept 

• Need to make clear what context LD's can be used in 

• Search for learning design used by this Learning object and vice versa. 

• Search by learning style (modality) 

• Search for LD based on Learning Object 

• While reading results was scared off by the domain in the results 

Features 

• Real world vs. Theoretical examples would be nice—i.e., case study vs. hands on complete problem 

• Would be nice to have an email sent that matches the criteria I requested, whenever the system is 
updated. 

• "A glossary would be useful" 

• Keyword search through education rationales 

• Grade level might be tied to duration 

• Will references be annotated (as compared to the LD's being annotated)? 

• Search by concept / topic / content 

Interface 

• Need to provide Google style Boolean keyword search 

• Search results need criteria as part of the title for human memory 

• Put the search button at the top of page as well 

• If you check more than one does it AND or OR? 

• Explanation needs to be more clear—should be geared towards the searcher not the student 

• Evidence—short record as default (Full record—should be available on demand) 

2. Comments and Suggestions Regarding Educational Rationale Descriptors 
Usage 

• If I had the list of ER tags in front of me, I would think using that model 

• Have the list of ER tags is reason to give a clue as to what should be entered 

Comments 

• There are a lot of Terms 

• Some tags appear to be more general than others 

• Is there a way to group ER tags? 

• Two tiers of ER would allow for refinement 

• ER tags—Terms are not defined 

• Watch for consistent use of nouns and verbs in ER tags 

• Most profs need practical solutions not familiar with pedagogy Under ER? 

o If lecture course f more interactive /variety 
o If language course f more specific point-based 
o Questionnaire—what does prof want to do? 

New Tags 

• Experiment 

• Practice 

• Tutorial 

• Debate 






• Analyze 

• What ER tags are there which relate to things without a 'right' answer? 

3. General Comments Relating to Learning Design 
Questions 

• Does evaluation included student input? 

• What about copyright issues? 

• Would the prof really think this way? 

• Class size; LO's are so small discrete—how to adopt for large class size? 

• Who is going to be creating / attaching the evaluation / evidence f questionnaire to get information 
from the instructor? 

Observations 

• This would be very helpful for new faculty 

• Think in the generic—not specific model 

• Need to remember to step back and think "how do I teach?" not just "how do I teach this?" 

• Evidence a good way for faculty to learn about the literature. 

• Would be easier to share learning designs than learning objects because you don't need multimedia 
skills to modify them. 

• Like discipline ideal for new profs and cross-discipline searching 

Suggestions 

• I would use this if it provided field specific concepts 

• Do a survey of what vocabulary faculty would use. 

• The 4 E's are quite forced. 

• Rename evidence to educational rationale 

• Bloom's taxonomy would be useful (secondary school teachers know this) 

• Need to provide faculty with a frame of reference, i.e., once they see an example, they can understand 
better 

• Delivery method f it's about learning not teaching designs the list should include independent 
learning 

• Suggested grade level, e.g., learning design suitable for all 1st years vs. only 4th years 

Misconceptions 

• I would not have thought of collaborate to deal with loops in computer science. 

• Thought LD's were small only—not semester long as well 

on the IMS Learning Design Specification 

During the process of transferring the selected learning designs into XML format we identified 
some apparent limitations within the IMS LD specification. We have by no means attempted to 
create an exhaustive list of recommendations, but felt the following two issues should be 
acknowledged. 

The first has to do with modeling services. Within the learning design examples we identified, 
there were certain service type activities that did not seem to be accommodated easily within 
the XML model. For example—it is unclear within the Learning Design XML specification how to 
indicate the properties and structure of a file uploader (who, how often, etcAA). 

It should be noted that within the IMS Learning Design Information Model (2003), under 
"services," it is explained that the service specification is extensible by namespacing in additional 
services. However, there are no examples to indicate how this would be done. 


The paper further outlines that for this version of the specification, the types of services specified 
are limiting these to those that are now found in typical LMS systems. It is possible to inherit 



from a generic service and thus specify new types as extensions to the vocabulary. As designers 
we would expect to see more instruction and or concrete examples of how to construct a 
"generic service." 

The four types of services provided in the IMS LD specification ( send-mailconference, monitor 
(level B), and index search ) represent a solid foundation, but as indicated in the specification 
document, the selection of services to be included needs to be driven by the community. 

The second apparent limitation we identified is related to capturing designer annotations directly 
within the Learning Design specification. Annotative descriptions are necessary to inform an end- 
user of the designer's intention or suggestions for instruction using the learning design. There is 
currently no area within the Learning Design specification for the designer to include critical 
comments or highlight features of the particular learning design. In the current specification, a 
full IMS meta-data record is required to provide a description. This is both cumbersome, and 
conceptually incorrect, as a meta-data record is intended to provide information about an object 
—whereas in this case, an annotative description is part of the object itself. 

We feel that designer annotations and comments would be useful for evaluating and 
disseminating information about the pedagogy surrounding a specific sequence of activities, 
service, environment or role, especially when interacting with the contents of a learning design 
repository. 

We can illustrate this point using our sample learning designs: Since there are no actual 
resources to point to, we found it difficult to provide a verbal description of an activity and the 
pedagogical rationale for each particular choice of activity and resource. As a work-round we 
have made use of the "title" tag to describing the generic activities—but this method seems like 
an abuse of the purpose of the title tag. We feel the ability to provide pedagogical and logistical 
annotations within Learning Design XML for each activity would be a great asset to the 
specification. 
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